A Reinforcement Learning Technique with an Adaptive Action Generator for a Multi-robot System

A robust instance-based reinforcement learning (RL) approach for controlling autonomous multi-robot systems (MRS) is introduced in this chapter. Although RL has been proven to be an effective approach for behavior acquisition for an autonomous robot, it generates considerably sensitive results for the segmentation of the state and action spaces. This problem can yield severe results with increase in the complexity of the system. When segmentation is inappropriate, RL often fails. Even if RL obtains successful results, the achieved behavior might not be sufficiently robust. In conventional RL, human designers segment the state and action spaces by using implicit knowledge based on their personal experience, because there are no guidelines for segmenting the state and action spaces. Two main approaches for solving the abovementioned problem and for learning in a continuous space have been discussed. One of the methods applies function-approximation techniques such as artificial neural networks to the Q-function. Sutton (Sutton, 1996) used CMAC and Morimoto and Doya (Morimoto & Doya, 2000) used Gaussian softmax basis functions for function approximation. Lin represented the Q-function by using multi-layer neural networks called Q-net (Lin, 1993). However, these techniques have the inherent difficulty that a human designer must properly design their neural networks before executing RL. The other method involves the adaptive segmentation of the continuous state space according to the robots' experiences. Asada et al. proposed a state clustering method based on the Mahalanobis distance (Asada et al., 1996). Takahashi et al. used the nearest-neighbor method (Takahashi et al., 1996). However, these methods generally require large learning costs for tasks such as the continuous update of data classifications every time new data arrives. Our research group has proposed an instance-based RL method called the continuous space classifier generator (CSCG), which proves to be effective for behavior acquisition (Svinin et al., 2000). We have also developed a second instance-based RL method called Bayesiandiscrimination-function-based reinforcement learning (BRL) (Yasuda et al., 2005). Our preliminary experiments proved that BRL, by means of adaptive segmentation of state and action spaces, exhibits better performance as compared to CSCG. As we mentioned in the previous chapter, BRL has an extended form that accelerates the learning speed (Yasuda & Ohkura, 2010). Our focal point for the extension is the process of action searching. In a standard BRL, a robot performs a random action and stores an input-